{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"jumbotron\">\n",
    "    <p class=\"display-1 h1\">注意力提示</p>\n",
    "    <hr class=\"my-4\">\n",
    "    <p>主讲：李岩</p>\n",
    "    <p>管理学院</p>\n",
    "    <p>liyan@cumtb.edu.cn</p>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 注意力的经济学视角\n",
    "\n",
    "### 注意力是一种稀缺资源\n",
    "\n",
    "**核心观点**：注意力是有限的、有价值的、稀缺的资源\n",
    "\n",
    "**为什么稀缺？**\n",
    "- 我们无法同时关注所有事物\n",
    "- 关注一件事意味着忽略其他事\n",
    "- 注意力有**机会成本**（就像金钱一样）\n",
    "\n",
    "**类比理解**：\n",
    "- 就像你选择读这本书，就不能同时读其他书\n",
    "- 就像你选择看一部电影，就不能同时看另一部\n",
    "- 注意力是有限的，需要明智地分配\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 注意力经济时代\n",
    "\n",
    "自经济学研究稀缺资源分配以来，人们正处在\"注意力经济\"时代，\n",
    "即人类的注意力被视为可以交换的、有限的、有价值的且稀缺的商品。\n",
    "\n",
    "**注意力经济的特征**：\n",
    "- 注意力被视为**可交换的商品**\n",
    "- 注意力是**有限的、有价值的、稀缺的**\n",
    "- 许多商业模式围绕注意力展开\n",
    "\n",
    "**实际例子**：\n",
    "- **流媒体服务**：看广告（消耗注意力）或付费（节省注意力）\n",
    "- **网络游戏**：花时间玩游戏（消耗注意力）或付费升级（节省注意力）\n",
    "- **社交媒体**：平台争夺用户的注意力\n",
    "\n",
    "**核心观点**：注意力不是免费的，需要付出代价。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 信息过载问题\n",
    "\n",
    "**人类面临的信息挑战**：\n",
    "- 视觉系统每秒接收约$10^8$位信息\n",
    "- 这**远远超过**大脑能完全处理的水平\n",
    "- 必须**选择性处理**信息\n",
    "\n",
    "**人类的解决方案**：\n",
    "- 从经验中学习：\"并非所有输入都同等重要\"\n",
    "- 将注意力集中在**感兴趣的一小部分信息**上\n",
    "- 忽略或降低其他信息的优先级\n",
    "\n",
    "**进化优势**：\n",
    "- 发现天敌（生存）\n",
    "- 找寻食物（生存）\n",
    "- 寻找伴侣（繁衍）\n",
    "- 社交互动（合作）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 生物学中的注意力提示\n",
    "\n",
    "注意力是如何应用于视觉世界中的呢？\n",
    "这要从当今十分普及的*双组件*（two-component）的框架开始讲起：\n",
    "这个框架的出现可以追溯到19世纪90年代的威廉·詹姆斯，\n",
    "他被认为是\"美国心理学之父\" :cite:`James.2007`。\n",
    "在这个框架中，受试者基于*非自主性提示*和*自主性提示*\n",
    "有选择地引导注意力的焦点。\n",
    "\n",
    "### 双组件框架\n",
    "\n",
    "**威廉·詹姆斯的贡献**（19世纪90年代）：\n",
    "- 被称为\"美国心理学之父\"\n",
    "- 提出了注意力的**双组件框架**\n",
    "\n",
    "**两种注意力提示**：\n",
    "1. **非自主性提示**：基于环境的突出性（自动的）\n",
    "2. **自主性提示**：基于认知和意识（主动的）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 非自主性提示\n",
    "\n",
    "非自主性提示是基于环境中物体的突出性和易见性。\n",
    "\n",
    "**定义**：基于环境中物体的**突出性和易见性**\n",
    "\n",
    "**例子场景**：\n",
    "想象一下，假如我们面前有五个物品：\n",
    "一份报纸、一篇研究论文、一杯咖啡、一本笔记本和一本书。\n",
    "所有纸制品都是黑白印刷的，但咖啡杯是红色的。\n",
    "换句话说，这个咖啡杯在这种视觉环境中是突出和显眼的，\n",
    "不由自主地引起人们的注意。\n",
    "\n",
    "**结果**：\n",
    "- 红色咖啡杯**自动吸引**注意力\n",
    "- 我们**不由自主**地看向咖啡杯\n",
    "- 这是**被动的、自动的**过程\n",
    "- 我们会把视力最敏锐的地方放到咖啡上\n",
    "\n",
    "**类比**：\n",
    "- 就像黑暗中突然亮起的灯\n",
    "- 就像人群中穿红色衣服的人\n",
    "- 突出的事物自动\"抓住\"我们的注意力\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![由于突出性的非自主性提示（红杯子），注意力不自主地指向了咖啡杯](../img/9_attention_mechanisms/eye-coffee.svg)\n",
    "\n",
    "**图1：非自主性注意力提示**\n",
    "\n",
    "红色咖啡杯因为其突出性，自动吸引了视觉注意力。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 自主性提示\n",
    "\n",
    "喝咖啡后，我们会变得兴奋并想读书，\n",
    "所以转过头，重新聚焦眼睛，然后看看书。\n",
    "与由于突出性导致的选择不同，\n",
    "此时选择书是受到了认知和意识的控制，\n",
    "因此注意力在基于自主性提示去辅助选择时将更为谨慎。\n",
    "受试者的主观意愿推动，选择的力量也就更强大。\n",
    "\n",
    "**定义**：基于**认知和意识**的主动选择\n",
    "\n",
    "**例子场景**：\n",
    "- 喝完咖啡后，**想要读书**\n",
    "- 主动**转过头**，**重新聚焦眼睛**\n",
    "- 看向书（而不是其他物品）\n",
    "\n",
    "**关键特点**：\n",
    "- 这是**主动的、有意识的**选择\n",
    "- 受到**主观意愿**驱动\n",
    "- 比非自主性提示**更强大、更精确**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![依赖于任务的意志提示（想读一本书），注意力被自主引导到书上](../img/9_attention_mechanisms/eye-book.svg)\n",
    "\n",
    "**图2：自主性注意力提示**\n",
    "\n",
    "基于主观意愿（想读书），注意力被主动引导到书上。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 两种提示的协同作用\n",
    "\n",
    "**实际应用中**：\n",
    "- 两种提示**共同工作**\n",
    "- 非自主性提示：快速筛选重要信息\n",
    "- 自主性提示：精确选择目标信息\n",
    "\n",
    "**例子**：\n",
    "- 开车时：红色信号灯（非自主性）自动吸引注意\n",
    "- 然后：主动查看（自主性）确认是否可以通行\n",
    "\n",
    "**重要性**：\n",
    "- 非自主性：提高效率，快速响应\n",
    "- 自主性：精确控制，实现目标\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 从生物学到神经网络：查询、键和值（QKV）\n",
    "\n",
    "### 引入QKV框架\n",
    "\n",
    "自主性的与非自主性的注意力提示解释了人类的注意力的方式，\n",
    "下面来看看如何通过这两种注意力提示，\n",
    "用神经网络来设计注意力机制的框架。\n",
    "\n",
    "**目标**：将人类注意力的机制应用到神经网络中\n",
    "\n",
    "**关键问题**：\n",
    "- 如何用数学表示非自主性提示？\n",
    "- 如何用数学表示自主性提示？\n",
    "- 如何设计注意力机制？\n",
    "\n",
    "**解决方案**：引入三个核心概念\n",
    "- **查询（Query，Q）**：自主性提示\n",
    "- **键（Key，K）**：非自主性提示\n",
    "- **值（Value，V）**：感官输入（实际信息）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### QKV框架的核心思想\n",
    "\n",
    "**QKV框架是注意力机制的基础**，理解这三个概念对于掌握注意力机制至关重要。\n",
    "\n",
    "**基本流程**：\n",
    "1. **查询（Q）**：表示\"我想要什么\"或\"我在寻找什么\"\n",
    "2. **键（K）**：表示\"我是什么\"或\"我的特征是什么\"\n",
    "3. **值（V）**：表示\"我包含的实际信息\"\n",
    "\n",
    "**工作方式**：\n",
    "- 查询与所有键比较，找到最匹配的键\n",
    "- 根据匹配程度，对对应的值进行加权\n",
    "- 最终输出是值的加权和\n",
    "\n",
    "**三者的关系图**：\n",
    "\n",
    "```\n",
    "输入信息\n",
    "   ↓\n",
    "[生成Q, K, V]\n",
    "   ↓\n",
    "Q ←→ K (匹配) → 注意力权重 → V (加权) → 输出\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**类比理解**：\n",
    "\n",
    "**1. 图书馆检索系统**：\n",
    "- **查询（Q）**：你想找的书名或主题（如\"深度学习\"）\n",
    "- **键（K）**：书籍的索引号、分类号（如\"TP181\"）\n",
    "- **值（V）**：书籍的实际内容（整本书）\n",
    "- **过程**：用查询匹配索引号，找到对应的书籍\n",
    "\n",
    "**2. 搜索引擎**：\n",
    "- **查询（Q）**：你的搜索关键词（如\"注意力机制\"）\n",
    "- **键（K）**：网页的关键词、标题、摘要\n",
    "- **值（V）**：网页的实际内容\n",
    "- **过程**：用查询匹配关键词，返回相关网页\n",
    "\n",
    "**3. 字典查找**：\n",
    "- **查询（Q）**：你想查的词（如\"attention\"）\n",
    "- **键（K）**：字典中的词条（按字母顺序排列）\n",
    "- **值（V）**：词条的定义和解释\n",
    "- **过程**：用查询匹配词条，找到对应的定义\n",
    "\n",
    "**关键理解**：\n",
    "- QKV框架将注意力机制**形式化**为数学操作\n",
    "- 这三个组件**缺一不可**，共同完成注意力计算\n",
    "- 理解QKV是理解所有注意力变体的基础"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 注意力机制 vs 传统方法\n",
    "\n",
    "首先，考虑一个相对简单的状况，\n",
    "即只使用非自主性提示。\n",
    "要想将选择偏向于感官输入，\n",
    "则可以简单地使用参数化的全连接层，\n",
    "甚至是非参数化的最大汇聚层或平均汇聚层。\n",
    "\n",
    "**传统方法（只使用非自主性提示）**：\n",
    "\n",
    "**实现方式**：\n",
    "- **全连接层**：参数化的，学习权重\n",
    "- **最大汇聚层**：非参数化的，选择最大值\n",
    "- **平均汇聚层**：非参数化的，计算平均值\n",
    "\n",
    "**特点**：\n",
    "- 所有输入**同等对待**（或基于固定规则）\n",
    "- 没有\"查询\"的概念\n",
    "- 选择是**固定的、预定义的**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 注意力机制的关键区别\n",
    "\n",
    "\n",
    "**核心区别**：\n",
    "- **是否包含自主性提示**（查询）\n",
    "\n",
    "**对比表格**：\n",
    "\n",
    "| 特性 | 全连接层/汇聚层 | 注意力机制 |\n",
    "|------|----------------|-----------|\n",
    "| **查询** | 无 | 有（自主性提示） |\n",
    "| **权重** | 固定 | 动态（根据查询变化） |\n",
    "| **选择性** | 无（或固定规则） | 有（自适应） |\n",
    "| **灵活性** | 低 | 高 |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 查询（Query，Q）\n",
    "\n",
    "**定义**：**自主性提示**，表示\"我想要什么\"或\"我在寻找什么\"\n",
    "\n",
    "**核心特点**：\n",
    "- 由**任务或目标**决定\n",
    "- 是**主动的、有意识的**\n",
    "- 可以**动态变化**\n",
    "- 代表**当前的需求或目标**\n",
    "\n",
    "**数学表示**：\n",
    "- 通常是一个向量：$\\mathbf{q} \\in \\mathbb{R}^{d_q}$\n",
    "- $d_q$是查询的维度\n",
    "\n",
    "**实际应用例子**：\n",
    "\n",
    "1. **机器翻译**：\n",
    "   - 查询：当前解码器要生成的目标词\n",
    "   - 例如：生成\"cat\"时，查询表示\"我需要找到与'cat'相关的源语言信息\"\n",
    "\n",
    "2. **图像描述生成**：\n",
    "   - 查询：当前要生成的词（如\"a\"、\"cat\"、\"sitting\"）\n",
    "   - 每个词对应不同的查询\n",
    "\n",
    "3. **问答系统**：\n",
    "   - 查询：用户的问题\n",
    "   - 例如：\"什么是注意力机制？\"\n",
    "\n",
    "4. **文本摘要**：\n",
    "   - 查询：当前要生成的摘要词\n",
    "   - 查询引导模型关注原文的哪些部分\n",
    "\n",
    "**作用**：\n",
    "- 指导注意力机制\"关注什么\"\n",
    "- 决定注意力权重的分配\n",
    "- 是注意力机制的\"驱动者\"\n",
    "\n",
    "**关键理解**：\n",
    "- 查询是**动态的**，每个时间步或每个任务都不同\n",
    "- 查询决定了注意力\"聚焦\"的方向\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 键（Key，K）\n",
    "\n",
    "**定义**：**非自主性提示**，表示输入信息的\"特征\"或\"标识\"\n",
    "\n",
    "**核心特点**：\n",
    "- 表示\"我是什么\"或\"我的特征是什么\"\n",
    "- 用于与查询进行**相似度匹配**\n",
    "- 是**静态的**（对于给定的输入）\n",
    "- 代表输入信息的**可检索特征**\n",
    "\n",
    "**数学表示**：\n",
    "- 通常是一个向量：$\\mathbf{k}_i \\in \\mathbb{R}^{d_k}$\n",
    "- $d_k$是键的维度\n",
    "- 可以有多个键：$\\mathbf{k}_1, \\mathbf{k}_2, \\ldots, \\mathbf{k}_m$\n",
    "\n",
    "**实际应用例子**：\n",
    "\n",
    "1. **机器翻译**：\n",
    "   - 键：源语言每个词的编码表示\n",
    "   - 例如：源句子\"I love cats\"中，每个词都有一个键\n",
    "\n",
    "2. **图像描述**：\n",
    "   - 键：图像的不同区域或特征\n",
    "   - 例如：图像中\"猫\"、\"椅子\"、\"窗户\"等区域的表示\n",
    "\n",
    "3. **问答系统**：\n",
    "   - 键：文档中每个句子或段落的表示\n",
    "   - 用于匹配问题\n",
    "\n",
    "**作用**：\n",
    "- 与查询比较，计算相似度\n",
    "- 决定哪些值应该被关注\n",
    "- 是注意力机制的\"匹配器\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 值（Value，V）\n",
    "\n",
    "**定义**：**感官输入**（实际信息），是我们真正想要获取的内容\n",
    "\n",
    "**核心特点**：\n",
    "- 包含**实际的信息内容**\n",
    "- 是我们最终要使用的数据\n",
    "- 可以是**完整的特征表示**\n",
    "- 与键**一一对应**\n",
    "\n",
    "**数学表示**：\n",
    "- 通常是一个向量：$\\mathbf{v}_i \\in \\mathbb{R}^{d_v}$\n",
    "- $d_v$是值的维度\n",
    "- 可以有多个值：$\\mathbf{v}_1, \\mathbf{v}_2, \\ldots, \\mathbf{v}_m$\n",
    "\n",
    "**实际应用例子**：\n",
    "\n",
    "1. **机器翻译**：\n",
    "   - 值：源语言每个词的完整编码（可能包含语义、语法等信息）\n",
    "   - 注意：值可能与键相同，也可能不同\n",
    "\n",
    "2. **图像描述**：\n",
    "   - 值：图像区域的完整特征表示\n",
    "   - 包含该区域的所有视觉信息\n",
    "\n",
    "3. **问答系统**：\n",
    "   - 值：文档中句子或段落的完整表示\n",
    "   - 包含回答问题的所有信息\n",
    "\n",
    "**作用**：\n",
    "- 提供实际的信息内容\n",
    "- 被注意力权重加权后形成输出\n",
    "- 是注意力机制的\"信息源\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 键值配对（Key-Value Pair）\n",
    "\n",
    "**重要概念**：每个值都有一个对应的键，形成键值对\n",
    "\n",
    "**配对关系**：\n",
    "- $(\\mathbf{k}_1, \\mathbf{v}_1), (\\mathbf{k}_2, \\mathbf{v}_2), \\ldots, (\\mathbf{k}_m, \\mathbf{v}_m)$\n",
    "- 键和值**一一对应**，但可以有不同的表示\n",
    "\n",
    "**为什么需要配对？**\n",
    "- 键用于**匹配**（与查询比较）\n",
    "- 值用于**输出**（实际信息）\n",
    "- 分离匹配逻辑和信息内容\n",
    "\n",
    "**类比**：\n",
    "- **字典**：键是索引（如\"attention\"），值是内容（如\"注意力的定义\"）\n",
    "- **数据库**：键是主键（如ID），值是记录（如完整数据）\n",
    "- **图书馆**：键是索引号，值是书籍\n",
    "\n",
    "### 键和值可以相同吗？\n",
    "\n",
    "**答案**：可以！这是常见的情况\n",
    "\n",
    "**自注意力（Self-Attention）**：\n",
    "- 查询、键、值都来自**同一个序列**\n",
    "- 例如：在Transformer中，Q、K、V都来自输入序列\n",
    "- 通过不同的线性投影得到不同的表示\n",
    "\n",
    "**键值分离的情况**：\n",
    "- 键：用于匹配的简化表示\n",
    "- 值：包含完整信息的表示\n",
    "- 例如：键可能是语义特征，值可能是完整的词向量\n",
    "\n",
    "**关键点**：\n",
    "- 键和值**可以相同**，也可以**不同**\n",
    "- 取决于具体的应用和设计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 为什么需要键值分离？\n",
    "\n",
    "**直接匹配值的问题**：\n",
    "- 值通常包含**完整的信息**，维度较高\n",
    "- 直接匹配计算量大\n",
    "- 匹配逻辑和信息内容**耦合**在一起\n",
    "- 不够灵活\n",
    "\n",
    "**键值分离的优势**：\n",
    "\n",
    "**1. 灵活性**\n",
    "- 键和值可以有不同的表示和维度\n",
    "- 键：可以是简化的特征（用于快速匹配）\n",
    "- 值：可以是完整的表示（包含所有信息）\n",
    "- 例如：键是128维，值是512维\n",
    "\n",
    "**2. 解耦设计**\n",
    "- **匹配逻辑**（键）和**信息内容**（值）分离\n",
    "- 可以独立优化匹配过程和信息表示\n",
    "- 更符合软件设计的\"关注点分离\"原则\n",
    "\n",
    "**3. 计算效率**\n",
    "- 键通常维度较小，匹配计算快\n",
    "- 可以预先计算键，动态匹配查询\n",
    "- 值只在需要时才参与计算\n",
    "\n",
    "**4. 表达能力**\n",
    "- 键可以专门学习\"如何被匹配\"\n",
    "- 值可以专门学习\"包含什么信息\"\n",
    "- 两者可以有不同的学习目标\n",
    "\n",
    "\n",
    "**关键理解**：\n",
    "- 键值分离是注意力机制的**核心设计**\n",
    "- 这种设计使得注意力机制既高效又灵活\n",
    "- 在Transformer中，Q、K、V通过不同的线性投影得到，体现了这种分离思想"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 注意力汇聚（Attention Pooling）\n",
    "\n",
    "**定义**：根据查询和键的匹配程度，计算权重，对值进行加权求和的过程\n",
    "\n",
    "**完整流程**：\n",
    "\n",
    "**步骤1：计算相似度分数**\n",
    "- 查询$\\mathbf{q}$与每个键$\\mathbf{k}_i$比较\n",
    "- 计算相似度：$s_i = \\text{similarity}(\\mathbf{q}, \\mathbf{k}_i)$\n",
    "- 得到分数向量：$\\mathbf{s} = [s_1, s_2, \\ldots, s_m]$\n",
    "\n",
    "**步骤2：转换为注意力权重**\n",
    "- 使用softmax归一化：$\\alpha_i = \\frac{\\exp(s_i)}{\\sum_{j=1}^m \\exp(s_j)}$\n",
    "- 得到权重向量：$\\boldsymbol{\\alpha} = [\\alpha_1, \\alpha_2, \\ldots, \\alpha_m]$\n",
    "- 权重满足：$\\sum_{i=1}^m \\alpha_i = 1$（概率分布）\n",
    "\n",
    "**步骤3：加权求和**\n",
    "- 用权重对值进行加权：$\\text{output} = \\sum_{i=1}^m \\alpha_i \\mathbf{v}_i$\n",
    "- 得到最终的注意力输出\n",
    "\n",
    "**数学形式化**：\n",
    "\n",
    "给定查询$\\mathbf{q}$和$m$个键值对$(\\mathbf{k}_1, \\mathbf{v}_1), \\ldots, (\\mathbf{k}_m, \\mathbf{v}_m)$：\n",
    "\n",
    "$$\\text{Attention}(\\mathbf{q}, \\mathbf{K}, \\mathbf{V}) = \\sum_{i=1}^m \\alpha_i \\mathbf{v}_i$$\n",
    "\n",
    "其中：\n",
    "$$\\alpha_i = \\frac{\\exp(a(\\mathbf{q}, \\mathbf{k}_i))}{\\sum_{j=1}^m \\exp(a(\\mathbf{q}, \\mathbf{k}_j))}$$\n",
    "\n",
    "$a(\\mathbf{q}, \\mathbf{k}_i)$是注意力评分函数。\n",
    "\n",
    "**关键特点**：\n",
    "- **动态权重**：权重根据查询动态变化\n",
    "- **选择性关注**：只关注相关的值\n",
    "- **软选择**：不是硬选择（0或1），而是概率分布\n",
    "\n",
    "**结果**：\n",
    "- 得到与查询最相关的信息\n",
    "- 权重高的值被更多地关注\n",
    "- 权重低的值被忽略或降低影响\n",
    "- 输出是值的**加权平均**，而不是简单平均\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![注意力机制通过注意力汇聚将*查询*（自主性提示）和*键*（非自主性提示）结合在一起，实现对*值*（感官输入）的选择倾向](../img/9_attention_mechanisms/qkv.svg)\n",
    ":label:`fig_qkv`\n",
    "\n",
    "**图3：查询、键、值的关系**\n",
    "\n",
    "**工作流程**：\n",
    "1. 查询（自主性）与键（非自主性）匹配\n",
    "2. 计算注意力权重\n",
    "3. 对值（感官输入）进行加权求和\n",
    "4. 得到与查询最相关的输出\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### QKV框架总结\n",
    "\n",
    "**QKV框架是注意力机制的标准框架**，理解这三个组件及其关系至关重要。\n",
    "\n",
    "**三者的关系**：\n",
    "\n",
    "```\n",
    "查询（Q） ←→ 键（K） → 注意力权重 → 值（V） → 输出\n",
    "   ↓           ↓                        ↓\n",
    "自主性提示  非自主性提示             实际信息\n",
    "```\n",
    "\n",
    "**工作流程**：\n",
    "1. **查询（Q）**：提出需求\"我想要什么\"\n",
    "2. **键（K）**：提供特征\"我是什么\"\n",
    "3. **匹配**：Q与K比较，计算相似度\n",
    "4. **权重**：相似度转换为注意力权重\n",
    "5. **值（V）**：提供实际信息\n",
    "6. **输出**：用权重对V加权求和\n",
    "\n",
    "**关键要点**：\n",
    "\n",
    "1. **Q决定关注方向**：\n",
    "   - 不同的Q会产生不同的注意力权重\n",
    "   - Q是动态的，随任务变化\n",
    "\n",
    "2. **K决定匹配方式**：\n",
    "   - K与Q的匹配方式决定了注意力模式\n",
    "   - K可以是静态的（输入固定）或动态的\n",
    "\n",
    "3. **V提供实际信息**：\n",
    "   - V是被关注的内容\n",
    "   - V的加权和形成最终输出\n",
    "\n",
    "4. **QKV可以相同**：\n",
    "   - 在自注意力中，Q、K、V都来自同一输入\n",
    "   - 通过不同的线性投影得到不同的表示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 文本处理中的QKV具体例子\n",
    "\n",
    "为了更好地理解QKV框架，让我们通过**文本处理**的具体例子来看Q、K、V在实际应用中是如何体现的。\n",
    "\n",
    "**为什么选择文本处理？**\n",
    "- 文本是序列数据，非常适合展示注意力机制\n",
    "- 例子直观易懂，便于理解\n",
    "- 是注意力机制的主要应用领域之一"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 例子1：机器翻译（编码器-解码器注意力）\n",
    "\n",
    "**任务**：将英文翻译成中文\n",
    "\n",
    "**输入句子（英文）**：\"I love cats\"\n",
    "\n",
    "**目标句子（中文）**：\"我喜欢猫\"\n",
    "\n",
    "**QKV的体现**：\n",
    "\n",
    "**查询（Q）**：\n",
    "- 来源：**解码器**当前要生成的目标词\n",
    "- 例子：生成\"喜欢\"时，查询表示\"我需要找到与'喜欢'相关的源语言信息\"\n",
    "- 特点：**动态的**，每个目标词对应不同的查询\n",
    "- 数学表示：$\\mathbf{q}_t$ 表示第$t$个时间步的查询\n",
    "\n",
    "**键（K）**：\n",
    "- 来源：**编码器**对源语言每个词的编码表示\n",
    "- 例子：\"I\"、\"love\"、\"cats\"每个词都有一个键\n",
    "- 特点：**静态的**（对于给定的源句子）\n",
    "- 数学表示：$\\mathbf{k}_1, \\mathbf{k}_2, \\mathbf{k}_3$ 分别对应\"I\"、\"love\"、\"cats\"\n",
    "\n",
    "**值（V）**：\n",
    "- 来源：**编码器**对源语言每个词的完整编码（通常与键相同）\n",
    "- 例子：\"I\"、\"love\"、\"cats\"每个词的完整语义表示\n",
    "- 特点：包含**实际的信息内容**\n",
    "- 数学表示：$\\mathbf{v}_1, \\mathbf{v}_2, \\mathbf{v}_3$ 分别对应\"I\"、\"love\"、\"cats\"\n",
    "\n",
    "**工作过程**：\n",
    "1. 生成\"喜欢\"时，查询$\\mathbf{q}_{喜欢}$与所有键比较\n",
    "2. 发现$\\mathbf{q}_{喜欢}$与$\\mathbf{k}_{love}$最匹配（注意力权重高）\n",
    "3. 使用高权重对$\\mathbf{v}_{love}$进行加权\n",
    "4. 得到与\"喜欢\"相关的信息，用于生成目标词"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 例子2：自注意力（Self-Attention）\n",
    "\n",
    "**任务**：理解句子中词与词之间的关系\n",
    "\n",
    "**输入句子**：\"The cat sat on the mat\"\n",
    "\n",
    "**QKV的体现**：\n",
    "\n",
    "**重要特点**：Q、K、V都来自**同一个序列**！\n",
    "\n",
    "**查询（Q）**：\n",
    "- 来源：输入序列中每个词的表示\n",
    "- 例子：\"cat\"的查询表示\"我想知道与'cat'相关的其他词\"\n",
    "- 特点：每个词都有自己的查询\n",
    "- 数学表示：$\\mathbf{q}_1, \\mathbf{q}_2, \\ldots, \\mathbf{q}_6$ 分别对应6个词\n",
    "\n",
    "**键（K）**：\n",
    "- 来源：输入序列中每个词的表示（与Q来源相同）\n",
    "- 例子：\"cat\"、\"sat\"、\"on\"等每个词都有键\n",
    "- 特点：用于与查询匹配\n",
    "- 数学表示：$\\mathbf{k}_1, \\mathbf{k}_2, \\ldots, \\mathbf{k}_6$\n",
    "\n",
    "**值（V）**：\n",
    "- 来源：输入序列中每个词的表示（与Q、K来源相同）\n",
    "- 例子：每个词的完整语义表示\n",
    "- 特点：包含实际信息\n",
    "- 数学表示：$\\mathbf{v}_1, \\mathbf{v}_2, \\ldots, \\mathbf{v}_6$\n",
    "\n",
    "**工作过程**：\n",
    "1. \"cat\"的查询$\\mathbf{q}_{cat}$与所有键比较\n",
    "2. 发现与$\\mathbf{k}_{sat}$匹配度高（因为\"cat sat\"是主谓关系）\n",
    "3. 使用注意力权重对$\\mathbf{v}_{sat}$加权\n",
    "4. \"cat\"能够\"看到\"\"sat\"的信息，理解语法关系\n",
    "\n",
    "**关键理解**：\n",
    "- 虽然Q、K、V都来自同一输入，但通过**不同的线性投影**得到不同的表示\n",
    "- 这使得每个词能够关注到序列中其他相关词的信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 例子3：问答系统（Question Answering）\n",
    "\n",
    "**任务**：根据文档回答问题\n",
    "\n",
    "**文档**：\"深度学习是机器学习的一个分支。它使用神经网络来学习数据的表示。\"\n",
    "\n",
    "**问题**：\"什么是深度学习？\"\n",
    "\n",
    "**QKV的体现**：\n",
    "\n",
    "**查询（Q）**：\n",
    "- 来源：**问题**的编码表示\n",
    "- 例子：\"什么是深度学习？\"的查询表示\"我想找到与问题相关的文档信息\"\n",
    "- 特点：**固定的**（对于给定问题）\n",
    "- 数学表示：$\\mathbf{q}_{问题}$\n",
    "\n",
    "**键（K）**：\n",
    "- 来源：**文档**中每个句子或段落的表示\n",
    "- 例子：文档中每个句子的编码\n",
    "- 特点：用于匹配问题\n",
    "- 数学表示：$\\mathbf{k}_1, \\mathbf{k}_2$ 分别对应两个句子\n",
    "\n",
    "**值（V）**：\n",
    "- 来源：**文档**中每个句子或段落的完整表示\n",
    "- 例子：每个句子的完整语义信息\n",
    "- 特点：包含答案信息\n",
    "- 数学表示：$\\mathbf{v}_1, \\mathbf{v}_2$\n",
    "\n",
    "**工作过程**：\n",
    "1. 问题的查询$\\mathbf{q}_{问题}$与文档中所有键比较\n",
    "2. 发现与第一个句子的键$\\mathbf{k}_1$匹配度高（因为包含\"深度学习\"的定义）\n",
    "3. 使用高权重对$\\mathbf{v}_1$进行加权\n",
    "4. 从第一个句子中提取答案：\"深度学习是机器学习的一个分支\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 例子4：文本摘要（Text Summarization）\n",
    "\n",
    "**任务**：生成原文的摘要\n",
    "\n",
    "**原文**：\"人工智能正在改变我们的生活。机器学习是人工智能的核心技术。深度学习是机器学习的一个分支。\"\n",
    "\n",
    "**QKV的体现**：\n",
    "\n",
    "**查询（Q）**：\n",
    "- 来源：**解码器**当前要生成的摘要词\n",
    "- 例子：生成\"人工智能\"时，查询表示\"我需要找到原文中与'人工智能'相关的信息\"\n",
    "- 特点：**动态的**，每个摘要词对应不同的查询\n",
    "- 数学表示：$\\mathbf{q}_t$ 表示第$t$个摘要词的查询\n",
    "\n",
    "**键（K）**：\n",
    "- 来源：**编码器**对原文每个词的编码\n",
    "- 例子：原文中每个词的表示\n",
    "- 特点：用于匹配查询\n",
    "- 数学表示：$\\mathbf{k}_1, \\mathbf{k}_2, \\ldots, \\mathbf{k}_n$\n",
    "\n",
    "**值（V）**：\n",
    "- 来源：**编码器**对原文每个词的完整编码\n",
    "- 例子：每个词的完整语义信息\n",
    "- 特点：包含实际信息\n",
    "- 数学表示：$\\mathbf{v}_1, \\mathbf{v}_2, \\ldots, \\mathbf{v}_n$\n",
    "\n",
    "**工作过程**：\n",
    "1. 生成摘要词时，查询与原文所有键比较\n",
    "2. 找到最相关的原文词（注意力权重高）\n",
    "3. 使用权重对对应的值进行加权\n",
    "4. 生成摘要词，确保摘要包含原文的关键信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### QKV示例总结\n",
    "\n",
    "**关键观察**：\n",
    "\n",
    "| 应用场景 | Q的来源 | K的来源 | V的来源 | QKV关系 |\n",
    "|---------|--------|--------|--------|---------|\n",
    "| **机器翻译** | 解码器（目标词） | 编码器（源词） | 编码器（源词） | Q≠K=V |\n",
    "| **自注意力** | 输入序列 | 输入序列 | 输入序列 | Q=K=V（同源） |\n",
    "| **问答系统** | 问题 | 文档 | 文档 | Q≠K=V |\n",
    "| **文本摘要** | 解码器（摘要词） | 编码器（原文词） | 编码器（原文词） | Q≠K=V |\n",
    "\n",
    "**重要理解**：\n",
    "\n",
    "1. **Q总是表示\"我想要什么\"**：\n",
    "   - 在生成任务中，Q是当前要生成的内容\n",
    "   - 在理解任务中，Q是当前要理解的内容\n",
    "\n",
    "2. **K总是表示\"我是什么\"**：\n",
    "   - K用于与Q匹配，计算相似度\n",
    "   - K可以是静态的（输入固定）或动态的\n",
    "\n",
    "3. **V总是表示\"实际信息\"**：\n",
    "   - V是被关注的内容\n",
    "   - 通常K和V相同（来自同一源），但可以不同\n",
    "\n",
    "4. **QKV可以相同也可以不同**：\n",
    "   - 自注意力：Q=K=V（都来自同一输入）\n",
    "   - 交叉注意力：Q≠K=V（Q来自一个序列，K和V来自另一个序列）\n",
    "\n",
    "**实际应用中的维度**：\n",
    "- 假设每个词用512维向量表示\n",
    "- Q的形状：$(n_q, 512)$，其中$n_q$是查询数量\n",
    "- K的形状：$(n_k, 512)$，其中$n_k$是键数量\n",
    "- V的形状：$(n_v, 512)$，其中$n_v$是值数量（通常$n_k = n_v$）\n",
    "- 注意力权重：$(n_q, n_k)$，表示每个查询对每个键的注意力"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 注意力的可视化\n",
    "\n",
    "### 为什么需要可视化？\n",
    "\n",
    "平均汇聚层可以被视为输入的加权平均值，\n",
    "其中各输入的权重是一样的。\n",
    "实际上，注意力汇聚得到的是加权平均的总和值，\n",
    "其中权重是在给定的查询和不同的键之间计算得出的。\n",
    "\n",
    "**可视化的重要性**：\n",
    "- 理解模型\"关注什么\"\n",
    "- 调试注意力机制\n",
    "- 解释模型行为\n",
    "- 验证注意力是否合理\n",
    "\n",
    "**常见可视化方式**：\n",
    "- **热图（Heatmap）**：用颜色深浅表示权重\n",
    "- **注意力权重矩阵**：显示查询-键的匹配关系\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 平均汇聚 vs 注意力汇聚\n",
    "\n",
    "**平均汇聚层**：\n",
    "- 所有输入的权重**相同**\n",
    "- 简单平均：$\\frac{1}{n}\\sum_{i=1}^{n} x_i$\n",
    "- 不考虑查询，对所有输入一视同仁\n",
    "\n",
    "**注意力汇聚**：\n",
    "- 输入的权重**不同**\n",
    "- 权重由查询和键的匹配程度决定\n",
    "- 加权平均：$\\sum_{i=1}^{n} \\alpha_i x_i$，其中$\\alpha_i$是注意力权重\n",
    "\n",
    "**关键区别**：\n",
    "- 平均汇聚：固定权重\n",
    "- 注意力汇聚：**动态权重**，根据查询变化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:03:03.521946Z",
     "iopub.status.busy": "2023-08-18T07:03:03.521507Z",
     "iopub.status.idle": "2023-08-18T07:03:05.621623Z",
     "shell.execute_reply": "2023-08-18T07:03:05.620583Z"
    },
    "origin_pos": 2,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 5,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "为了可视化注意力权重，需要定义一个`show_heatmaps`函数。\n",
    "其输入`matrices`的形状是\n",
    "（要显示的行数，要显示的列数，查询的数目，键的数目）。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:03:05.627152Z",
     "iopub.status.busy": "2023-08-18T07:03:05.626530Z",
     "iopub.status.idle": "2023-08-18T07:03:05.634951Z",
     "shell.execute_reply": "2023-08-18T07:03:05.633763Z"
    },
    "origin_pos": 6,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def show_heatmaps(matrices, xlabel, ylabel, titles=None, figsize=(2.5, 2.5),\n",
    "                  cmap='Reds'):\n",
    "    \"\"\"显示矩阵热图\"\"\"\n",
    "    d2l.use_svg_display()\n",
    "    num_rows, num_cols = matrices.shape[0], matrices.shape[1]\n",
    "    fig, axes = d2l.plt.subplots(num_rows, num_cols, figsize=figsize,\n",
    "                                 sharex=True, sharey=True, squeeze=False)\n",
    "    for i, (row_axes, row_matrices) in enumerate(zip(axes, matrices)):\n",
    "        for j, (ax, matrix) in enumerate(zip(row_axes, row_matrices)):\n",
    "            pcm = ax.imshow(matrix.detach().numpy(), cmap=cmap)\n",
    "            if i == num_rows - 1:\n",
    "                ax.set_xlabel(xlabel)\n",
    "            if j == 0:\n",
    "                ax.set_ylabel(ylabel)\n",
    "            if titles:\n",
    "                ax.set_title(titles[j])\n",
    "    fig.colorbar(pcm, ax=axes, shrink=0.6);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 7,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "下面使用一个简单的例子进行演示。\n",
    "在本例子中，仅当查询和键相同时，注意力权重为1，否则为0。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T07:03:05.640096Z",
     "iopub.status.busy": "2023-08-18T07:03:05.639355Z",
     "iopub.status.idle": "2023-08-18T07:03:05.886353Z",
     "shell.execute_reply": "2023-08-18T07:03:05.885235Z"
    },
    "origin_pos": 8,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"193.35825pt\" height=\"156.35625pt\" viewBox=\"0 0 193.35825 156.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-08-18T07:03:05.823629</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M -0 156.35625 \n",
       "L 193.35825 156.35625 \n",
       "L 193.35825 0 \n",
       "L -0 0 \n",
       "L -0 156.35625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 34.240625 118.8 \n",
       "L 145.840625 118.8 \n",
       "L 145.840625 7.2 \n",
       "L 34.240625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g clip-path=\"url(#p3a36c213e6)\">\n",
       "    <image xlink:href=\"data:image/png;base64,\n",
       "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAYAAADG4PRLAAABc0lEQVR4nO3dsW3CUBhGURPRJSNkPxgpA2YAGmqnpfRDMnnXnFO7sHT1N58s+bTeb+vCVK6f35uf/djxPXgBAeMEjBMwTsA4AeMEjBMwTsA4AePO//0C72JkHvu5/25+1gXGCRgnYJyAcQLGCRgnYJyAcQLGCRhnSnvSyDS2LGPz2AgXGCdgnIBxAsYJGCdgnIBxAsYJGCdgnIBxttAHe336tycXGCdgnIBxAsYJGCdgnIBxAsYJGCdg3OGntOI8NsIFxgkYJ2CcgHECxgkYJ2CcgHECxgkYl5zSjj6PjXCBcQLGCRgnYJyAcQLGCRgnYJyAcQLGTTOlmcee4wLjBIwTME7AOAHjBIwTME7AOAHjBIzbbUqb5b8KR+cC4wSMEzBOwDgB4wSMEzBOwDgB4wSMEzBuaAv16d98XGCcgHECxgkYJ2CcgHECxgkYJ2CcgHGny/K1bn3YPDYfFxgnYJyAcQLGCRgnYJyAcQLGCRgnYNwfkEghRAiKZdAAAAAASUVORK5CYII=\" id=\"image8a96b358df\" transform=\"scale(1 -1)translate(0 -112)\" x=\"34.240625\" y=\"-6.8\" width=\"112\" height=\"112\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"mdcf2e5a8c4\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdcf2e5a8c4\" x=\"39.820625\" y=\"118.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(36.639375 133.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdcf2e5a8c4\" x=\"95.620625\" y=\"118.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(92.439375 133.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_3\">\n",
       "     <!-- Keys -->\n",
       "     <g transform=\"translate(78.371094 147.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-4b\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2694 \n",
       "L 3353 4666 \n",
       "L 4166 4666 \n",
       "L 1850 2491 \n",
       "L 4331 0 \n",
       "L 3500 0 \n",
       "L 1259 2247 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-4b\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"60.576172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"122.099609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"181.279297\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <defs>\n",
       "       <path id=\"m473666877a\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m473666877a\" x=\"34.240625\" y=\"12.78\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(20.878125 16.579219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m473666877a\" x=\"34.240625\" y=\"35.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(20.878125 38.899219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m473666877a\" x=\"34.240625\" y=\"57.42\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(20.878125 61.219219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m473666877a\" x=\"34.240625\" y=\"79.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(20.878125 83.539219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m473666877a\" x=\"34.240625\" y=\"102.06\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(20.878125 105.859219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- Queries -->\n",
       "     <g transform=\"translate(14.798437 82.307031)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-51\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 3406 84 \n",
       "L 4238 -825 \n",
       "L 3475 -825 \n",
       "L 2784 -78 \n",
       "Q 2681 -84 2626 -87 \n",
       "Q 2572 -91 2522 -91 \n",
       "Q 1538 -91 948 567 \n",
       "Q 359 1225 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1516 4351 937 \n",
       "Q 4025 359 3406 84 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"78.710938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"142.089844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"203.613281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"244.726562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"272.509766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"334.033203\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 34.240625 118.8 \n",
       "L 34.240625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 145.840625 118.8 \n",
       "L 145.840625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 34.240625 118.8 \n",
       "L 145.840625 118.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 34.240625 7.2 \n",
       "L 145.840625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 152.815625 103.77 \n",
       "L 156.892625 103.77 \n",
       "L 156.892625 22.23 \n",
       "L 152.815625 22.23 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path clip-path=\"url(#p81ac4143b7)\" style=\"fill: #ffffff; stroke: #ffffff; stroke-width: 0.01; stroke-linejoin: miter\"/>\n",
       "   </g>\n",
       "   <image xlink:href=\"data:image/png;base64,\n",
       "iVBORw0KGgoAAAANSUhEUgAAAAQAAABRCAYAAAD1sgc6AAAAnklEQVR4nJ2Suw7CQAwEjZT//1QqCnR+0XKzJxmSLqPx7krJo1/Ptq/nsgzbQRVBDqBH4wCkhTtkWDhqR8OSIHjii6H/Z2jtaLD2EDpm3DCkNvldftgB0GJIqM/TBfAfE+AAJcaSpZJRg1Fs0QyCjN5BRA0gkycEmTyRFhox7simsb/blSbGDWCDcdgB4MxwGu8CWAJ4shgqJ9IiBms/jwXJt9gA8G4AAAAASUVORK5CYII=\" id=\"image52587ed27e\" transform=\"scale(1 -1)translate(0 -81)\" x=\"153\" y=\"-22\" width=\"4\" height=\"81\"/>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <defs>\n",
       "       <path id=\"m4194a474b4\" d=\"M 0 0 \n",
       "L 3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4194a474b4\" x=\"156.892625\" y=\"103.77\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.00 -->\n",
       "      <g transform=\"translate(163.892625 107.569219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4194a474b4\" x=\"156.892625\" y=\"83.385\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(163.892625 87.184219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4194a474b4\" x=\"156.892625\" y=\"63\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.50 -->\n",
       "      <g transform=\"translate(163.892625 66.799219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4194a474b4\" x=\"156.892625\" y=\"42.615\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.75 -->\n",
       "      <g transform=\"translate(163.892625 46.414219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n",
       "L 3525 4666 \n",
       "L 3525 4397 \n",
       "L 1831 0 \n",
       "L 1172 0 \n",
       "L 2766 4134 \n",
       "L 525 4134 \n",
       "L 525 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-37\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4194a474b4\" x=\"156.892625\" y=\"22.23\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 1.00 -->\n",
       "      <g transform=\"translate(163.892625 26.029219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\"/>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 152.815625 103.77 \n",
       "L 154.854125 103.77 \n",
       "L 156.892625 103.77 \n",
       "L 156.892625 22.23 \n",
       "L 154.854125 22.23 \n",
       "L 152.815625 22.23 \n",
       "L 152.815625 103.77 \n",
       "z\n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p3a36c213e6\">\n",
       "   <rect x=\"34.240625\" y=\"7.2\" width=\"111.6\" height=\"111.6\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p81ac4143b7\">\n",
       "   <rect x=\"152.815625\" y=\"22.23\" width=\"4.077\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 180x180 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "attention_weights = torch.eye(10).reshape((1, 1, 10, 10))\n",
    "show_heatmaps(attention_weights, xlabel='Keys', ylabel='Queries')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 总结与思考\n",
    "\n",
    "### 关键概念回顾\n",
    "\n",
    "1. **注意力的本质**：\n",
    "   - 注意力是稀缺资源\n",
    "   - 需要选择性关注重要信息\n",
    "   - 有经济学和生物学基础\n",
    "\n",
    "2. **两种注意力提示**：\n",
    "   - 非自主性提示：基于突出性，被动\n",
    "   - 自主性提示：基于意识，主动\n",
    "\n",
    "3. **查询-键-值框架**：\n",
    "   - 查询（Q）：自主性提示\n",
    "   - 键（K）：非自主性提示\n",
    "   - 值（V）：感官输入\n",
    "\n",
    "4. **注意力机制的特点**：\n",
    "   - 包含自主性提示（查询）\n",
    "   - 动态权重分配\n",
    "   - 与全连接层/汇聚层不同\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 注意力机制 vs 传统方法对比\n",
    "\n",
    "| 特性 | 全连接层/汇聚层 | 注意力机制 |\n",
    "|------|----------------|-----------|\n",
    "| **查询** | 无 | 有（自主性提示） |\n",
    "| **权重** | 固定 | 动态（根据查询变化） |\n",
    "| **选择性** | 无（或固定规则） | 有（自适应） |\n",
    "| **灵活性** | 低 | 高 |\n",
    "\n",
    "**关键区别**：\n",
    "- 注意力机制能够**根据查询动态调整**关注点\n",
    "- 传统方法对所有输入**一视同仁**或使用**固定规则**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 思考问题\n",
    "\n",
    "1. **为什么注意力是稀缺的？**\n",
    "   - 从信息处理的角度解释\n",
    "   - 从经济学的角度解释\n",
    "\n",
    "2. **非自主性和自主性提示的区别是什么？**\n",
    "   - 它们如何协同工作？\n",
    "   - 在神经网络中如何表示？\n",
    "\n",
    "3. **为什么需要键值分离？**\n",
    "   - 键和值可以相同吗？\n",
    "   - 分离有什么好处？\n",
    "\n",
    "4. **注意力机制如何解决信息瓶颈问题？**\n",
    "   - 与编码器-解码器架构的关系是什么？\n",
    "\n",
    "5. **如何理解注意力权重？**\n",
    "   - 权重为1和权重为0.5分别意味着什么？\n",
    "   - 为什么需要softmax归一化？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 练习\n",
    "\n",
    "1. **机器翻译中的注意力**：\n",
    "   - 在机器翻译中，解码序列词元时：\n",
    "     - 自主性提示（查询）可能是什么？\n",
    "     - 非自主性提示（键）可能是什么？\n",
    "     - 感官输入（值）可能是什么？\n",
    "   - 提示：考虑编码器-解码器架构\n",
    "\n",
    "2. **注意力权重可视化**：\n",
    "   - 随机生成一个$10 \\times 10$矩阵\n",
    "   - 使用`softmax`运算确保每行都是有效的概率分布\n",
    "   - 使用`show_heatmaps`可视化注意力权重\n",
    "   - 观察权重分布的特点\n",
    "\n",
    "3. **注意力机制的应用**：\n",
    "   - 除了机器翻译，注意力机制还能用于哪些任务？\n",
    "   - 在这些任务中，查询、键、值分别是什么？\n",
    "\n",
    "4. **注意力权重的理解**：\n",
    "   - 如果注意力权重矩阵是对角矩阵，意味着什么？\n",
    "   - 如果注意力权重矩阵是均匀矩阵，意味着什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 下一步学习\n",
    "\n",
    "- **注意力汇聚**：如何计算注意力权重\n",
    "- **注意力评分函数**：如何衡量查询和键的相似度\n",
    "- **Bahdanau注意力**：在RNN中的应用\n",
    "- **多头注意力**：同时使用多个查询\n",
    "- **Transformer**：完全基于注意力的架构\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 9,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "后面的章节内容将经常调用`show_heatmaps`函数来显示注意力权重。\n",
    "\n",
    "## 小结\n",
    "\n",
    "* 人类的注意力是有限的、有价值和稀缺的资源。\n",
    "* 受试者使用非自主性和自主性提示有选择性地引导注意力。前者基于突出性，后者则依赖于意识。\n",
    "* 注意力机制与全连接层或者汇聚层的区别源于增加的自主提示。\n",
    "* 注意力机制通过注意力汇聚使选择偏向于值（感官输入），其中包含查询（自主性提示）和键（非自主性提示）。键和值是成对的。\n",
    "* 可视化查询和键之间的注意力权重是可行的。\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "required_libs": [],
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
